MEMORANDUM January 23, 2015 
TO: Board Members 


FROM: Terry B. Grier, Ed.D. 
Superintendent of Schools 


SUBJECT: 2014 EVERYDAY EXCELLENCE INSTITUTE EVALUATION REPORT 
CONTACT: Carla Stevens, 713-556-6700 


English language learners (ELLs) are at risk of falling behind academically because of 
unaddressed gaps in academic language and literacy skills. The Everyday ExcELLence 
Institute was a collaboration between the Multilingual, Curriculum, Instruction, & 
Assessment, and Professional Support & Development departments and was intended 
to provide ELL teachers with practical instructional routines that could be used in a variety 
of content areas. 


The report summarizes data from the ExcELLence Institute training for teachers which 
occurred in 2013-2014. Included are demographic data for program participants, 
information on teacher reactions to the training and on their implementation of the 
strategies they learned, as well as data on the impact of training on the academic 
performance of students of those teachers. 


A total of 455 teachers attended the Everyday ExcELLence Institute, teaching in the areas 
of reading/ELA, mathematics, science, or social studies. Results showed that teachers 
were satisfied overall with the quality of the training. Teachers reported using most 
ExcELLence Institute strategies fairly frequently, but did express concern over the amount 
of ongoing support they had available, particularly from principals and others 
administrators. Finally, performance of ESL students whose teachers received training 
showed only weak evidence for beneficial effects compared to ESL students whose 
teachers did not receive training. However, there was stronger evidence for benefits of 
training when campuses had evidence of administrative buy-in and an ongoing focus on 
the use of the trained strategies. 


cS / 


“ oe. 


cc: Superintendent’s Direct Reports Chief School Officers 
Gracie Guerrero School Support Officers 
Jennifer Alexander 
Terrie Armstrong 


ee, — 


HOUSTON INDEPENDENT SCHOOL DISTRICT 


RESEARCH 


Educational Program Report 


EVERYDAY EXCELLENCE INSTITUTE 


ALUATION 


AFrAD rr A 
2013 - 2014 


Research and Accountability 


ANALY ZING DATA, MEASURING PERFORMANCE. 


HISD 


Houston Independent School District 
Hattie Mae White Educational Support Center 
4400 West 18th StreetHouston, Texas 77092-8501 


www.HoustonliSD.org 


It is the policy of the Houston Independent School 
District not to discriminate on the basis of age, color, 
handicap or disability, ancestry, national origin, 
marital status, race, religion, sex, veteran status, 
political affiliation, sexual orientation, gender identity 
and/or gender expression in its educational or 
employment programs and activities. 


2015 BOARD OF EDUCATION 


Rhonda Skillern-Jones 
President 


Manuel Rodriguez, Jr. 
First Vice President 


Wanda Adams 
Second Vice President 


Paula Harris 
Secretary 


Juliet Stipeche 
Assistant Secretary 


Anna Eastman 
Michael L. Lunceford 
Greg Meyers 

Harvin C. Moore 


Terry B. Grier, Ed.D. 
Superintendent of Schools 


Carla Stevens 
Assistant Superintendent 
Department of Research and Accountability 


Kevin Briand, Ph.D. 
Research Specialist 


Venita Holmes, Dr.P.H. 
Research Manager 


EVERYDAY EXCELLENCE INSTITUTE PROGRAM EVALUATION REPORT 
2013-2014 


Executive Summary 


Program Description 


More than 60,000 students in Houston ISD are labeled as “English language learners”, or ELLs. Many of 
these students have unaddressed gaps in academic language and literacy skills. Without proper instruc- 
tional supports, these students are at risk of falling behind academically. The Everyday ExcELLence 
Institute is a training session for teachers of ELL students, and has been offered since 2012. For the 
2013-2014 school year, the sessions were offered over a five-month period starting in June of 2013. 
There were two separate four-day sessions in June and July, as well as two one-day sessions in Sep- 
tember and October of 2013. The Institute was the product of a joint collaboration between the Multilin- 
gual, Curriculum, Instruction, & Assessment, and Professional Support and Development (PSD) depart- 
ments of HISD, and was aimed at teachers in grades one through twelve. 


The intent of the Everyday ExcELLence Institute was to provide teachers with practical instructional rou- 
tines that could be used with ELL students in any content area. Attendees were exposed to skills and 
practices that should allow them to better reach and engage their ELL students. These practices fell into 
two main categories. First, participants at the ExcELLence institute received training on sheltered in- 
struction. In addition, teachers were instructed in the use of eight literacy routines. Literacy and lan- 
guage acquisition research suggests that simple, high-impact instructional actions can help ELLs learn 
more new content while developing stronger vocabulary and literacy skills. The everyday excellence rou- 
tines were intended to be used daily, and are summarized on the PSD website at http:// 


houstonisdpsd.org/literacy-routines.html. 


The sheltered instruction portion of the training was delivered over the first two days of the four-day 
training sessions, while the eight literacy routines were reviewed on the final two days. In addition, dur- 
ing these final two days teachers had to create a lesson plan utilizing these newly acquired techniques 
(the schedule was adjusted accordingly for participants in the one-day Sessions). Training was provided 
by a team of district staff (Multilingual department staff and Teacher Development Specialists) who had 
themselves been trained on the various techniques covered. 


Highlights 


e A total of 455 teachers participated in the Everyday ExcELLence Institute, with 380 of them partici- 
pating for the full four days (24 hours) and an additional 75 attending for less than four days. 


e Institute participants, compared to other HISD teachers, were slightly older, had more teaching ex- 
perience, and were more likely to be African American. 


e One hundred and seven participants completed an online survey regarding their reactions to the 
training sessions, and their use of strategies learned while attending. 


e Overall, there was a high degree of satisfaction with the training, with 88% of teacher responses 
being positive. 


e Teachers were relatively less positive when asked to comment on their use of specific strategies 
during the school year. Some strategies were more difficult to use than others, and teachers also 
expressed concern over the support they received from administrators and other district staff. 


e« Performance of ELL students whose teachers received ExcELLence Institute training showed only 
weak evidence for beneficial effects, compared to those of other ELL students. 


e« Those benefits which were seen depended on both teachers’ core area (reading teachers had a 
lesser impact than did social studies teachers) and on the subject area tested. 


e There was stronger evidence for benefits of training when student data were analyzed for three 
campuses identified as having administrative buy-in and an ongoing focus on use of the strategies. 


Recommendations 


1. Available data indicates that this program improves student performance only weakly unless there is 
evidence that it was implemented to a high degree. More effort should be directed at ensuring that 
there is campus-level support and buy-in from administrators. 


2. The generally weak benefits seen for subjects other than reading/language arts indicates that the 
strategies, even when used, are not useful across all content areas. This counterintuitive finding 
should be investigated further by program staff to determine whether content-are teachers are im- 
plementing the strategies adequately. 


3. Broadening the program to include elementary as well as secondary students, and to target teach- 
ers of non-ELLs, suggests that the program may have lost focus. Irregardless of the potential bene- 
fits of this training to teachers of students at all levels, the original need (literacy issues with second- 
ary ELLs) may no longer be being met. Program staff should determine whether this program still 
addresses the needs of these students, and if necessary develop a program that targets them. 


Administrative Response 


One school, Hartman MS, showed a very high level of implementation. The principal, an immigrant him- 
self, is keenly aware of the needs of ELLs. He supported the implementation of this program at Hartman 
and also agreed to host summer trainings on his campus. In doing so, it is evident that a focus on meet- 
ing the needs of ELLs produced results (See p. 10). 


This project is now in the third year of implementation. At this time, the training has been divided up to 
offer the training during the school year. The first two days of the training, Sheltered Instruction Plus, is 
offered through the Multilingual Programs Department. The literacy routines have been divided in four 
sessions of two routines each. These sessions have been offered during district-wide early dismissal 
days. Very few teachers are receiving the four-day training as it was intended. Further changes have 
included an update of the eight literacy strategies with a focus of literacy development in general. In es- 
sence, the initiative has lost focus on the instructional needs of ELLs. 


The Multilingual Department is now looking at other sheltered instruction training approaches. One such 
training may be piloted this spring, and is based on the work of Dr. Eugenie Mora-Flores. 


Introduction 


There are more than 60,000 students in Houston ISD labeled as “English language learners”, or ELLs. 
Many of these students have unaddressed gaps in academic language and literacy skills. Without prop- 
er instructional supports, these students are at risk of falling behind academically. The Everyday ExcEL- 
Lence Institute is a training session for teachers of ELL students, and has been offered since 2012. For 
the 2013-2014 school year the sessions were offered over a five-month period starting in June of 2013. 
There were two separate four-day sessions in June and July, as well as two one-day sessions in Sep- 
tember and October of 2013. The Institute was the product of a joint collaboration between the Multilin- 
gual, Curriculum, Instruction, & Assessment, and Professional Support and Development (PSD) depart- 
ments of HISD, and was aimed at teachers in grades one through twelve. 


The intent of the Everyday ExcELLence Institute was to provide teachers with practical instructional rou- 
tines that could be used with ELL students in any content area. Attendees were exposed to skills and 
practices that should allow them to better reach and engage their ELL students. These practices fell into 
two main categories. First, participants at the ExcELLence institute received training on sheltered in- 
struction. Sheltered instruction training promotes and enhances the use of instructional strategies and 
modifications that allow ELLs to access an English language curriculum more effectively. This practical 
approach to "sheltering" English language learners emphasizes giving students the support they need to 
learn difficult new content while learning academic language. In addition, teachers were instructed in the 
use of eight literacy routines. Literacy and language acquisition research suggests that simple, high- 
impact instructional actions can help ELLs learn more new content while developing stronger vocabulary 
and literacy skills. The everyday excellence routines were intended to be used daily, and are summa- 


rized on the PSD website at http://houstonisdpsd.org/literacy-routines.html. 


The sheltered instruction portion of the training was delivered over the first two days of the four-day 
training sessions, while the eight literacy routines were reviewed on the final two days. In addition, dur- 
ing these final two days teachers had to create a lesson plan utilizing these newly acquired techniques 
(the schedule was adjusted accordingly for participants in the one-day Sessions). Training was provided 
by a team of district staff (Multilingual department staff and Teacher Development Specialists) who had 
themselves been trained on the various techniques covered. Appendix A (p. 13) summarizes the eight 
literacy routines, and Appendix B (p. 15) provides further background on sheltered instruction. 


Methods 


Participants 


A total of 455 teachers attended the Everyday ExcELLence Institute in 2013-2014. Most of these (380 
teachers, or 84%) received at least 24 hours of training, with an additional 50 (11% of teachers) receiv- 
ing at least 12 hours and 25 (5%) less than 12 hours (142 teachers registered for the training but did not 
attend). Appendix C (see p. 17) shows counts of teaching and non-teaching staff who attended training 
by campus. Student performance data were analyzed from all English as a Second Language (ESL) 
students who were in classes taught by teachers who attended the Everyday ExcELLence Institute in 
2013-2014. 


Data Collection & Analysis 


The Multilingual Department provided of a list of teachers attending the Everyday ExcELLence Institute, 
and teacher’s employee ID codes were retrieved from the district's Chancery database. Next, teacher 


demographic information was extracted from Chancery, including years of teaching experience. A list 
was created of all students in subjects taught by teachers who attended the training, which was then 
used to retrieve student performance data on various standardized tests. 


An online survey was used to collect data from teachers and other staff who attended the Everyday Ex- 
cELLence Institute 2013-2014. The first section of the survey sought feedback on attendees’ reactions 
to the training, what their experiences had been, what had worked, and what had not. A copy of the full 
survey, along with responses, is shown in Appendix D (p. 19). The second part of the survey con- 
cerned implementation of the various strategies they had learned. This section included questions about 
implementing these methods in the classroom, as well as questions concerning teachers’ use of specific 
strategies (Appendices E-G, pp. 20-22). Teachers completed the survey online at the end of the school 
year. Appendices D through G also include a summary of responses. 


Student performance data were collected from the State of Texas Assessments of Academic Readiness 
(STAAR and STAAR End-of-Course), Stanford Achievement Test (Stanford 10), and the Texas English 
Language Proficiency Assessment System (TELPAS). STAAR results are reported for the reading, 
mathematics, science, and social studies tests. For STAAR EOC, results were analyzed for Algebra I, 
Biology, English | and II, and U.S. History. Stanford 10 results are reported (Normal Curve Equivalents 
or NCEs) for reading, mathematics, language, science, and social science. 


TELPAS results are reported for two indicators. One indicator reflects the level of English language 
reading proficiency exhibited by ELL students (TELPAS reading scale score). The second indicator re- 
flects progress, i.e., whether students gained one or more levels of English language proficiency be- 
tween testing in 2013 and 2014. For this second TELPAS indicator, the percent of students gaining one 
or more proficiency levels in the previous year is reported. Appendix H (see p. 23) provides further de- 
tails on each of the assessments analyzed for this report. 


Results 
What was the demographic profile of teachers who received ExcELLence Institute training? 


e Eighty-three percent of teachers receiving training were female, compared to 75% for the district 
overall (significant difference at p<.05). 


e Everyday ExcELLence teachers were more likely to be African American, and less likely top be ei- 
ther Hispanic or White, compared to all district teachers (See Figure 1). 


Figure 1. Distribution of Everyday ExcELLence Institute trained teachers by ethnicity. 
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Figure 2. Distribution of Everyday ExcELLence Institute trained teachers by age. 
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Figure 2 shows the distribution of ages for teachers who received ExcELLence Institute training 
(shaded bars). Also included, for comparison, is the relative distribution of ages for teachers in the 
district (open circles). 


ExcELLence Institute teachers were on average slightly older than the average district teacher. The 
odds of being younger than 40 years old were lower for teachers who attended training than they 
were for other teachers in the district (odds ratio = 0.76, z = 2.77, p < .01, significant at p < .05). 


Figure 3. Distribution of trained teachers by years of previous experience teaching overall. 
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Figure 3 shows the distribution of prior experience (ExcELLence Institute teachers as bars, HISD 
overall as open circles). Trained teachers were no more likely to have one or fewer years of experi- 
ence than other teachers in the district (odds ratio = 0.85, z = 1.28, p>.10, not significant at p < .05). 


Figure 4 shows the distribution of teaching experience within HISD. ExcELLence Institute teachers 
were less likely to have one or fewer years of experience than other teachers in the district (odds 
ratio = 0.74, z = 2.64, p<.01, significant at p < .05). 


Figure 4. Distribution of trained teachers by years of previous experience teaching in HISD. 
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How Satisfied were teachers with the training received at the Everyday ExcELLence Institute? 


One hundred and seven individuals who had attended ExcELLence Institute training responded to an 
online survey assessing reaction to the training sessions (this was 24% of all those who attended). A full 
summary of responses to the entire survey can be found in Appendix D (p. 19). 


« Respondents were divided amongst elementary (40%), middle school (27%), and high school teach- 
ers (31%), with two non-teachers. Of the teacher respondents, 55% taught reading or English lan- 
guage arts, 34% taught math, 31% taught science, 33% taught social studies, and 13% taught other 
subjects. Note that the total is greater than 100% because some taught multiple subjects. 


e Overall, responses to this survey indicated a high degree of satisfaction with the training, with 88% 
of responses being positive. 


e The trainers: Opinions about the trainers were highly positive, with 90 percent or more either agree- 
ing or strongly agreeing with the following statements: “adequately set the tone and background for 
information presented” (90%), “allowed me to reflect and share my ideas/views about the topics pre- 
sented” (90%), "Helped me to make connections with the information so that | could use it in my 
teaching" (90%), and “actively encouraged collaborative discussion” (92%). 


e The training sessions: Statements which received the highest degree of support were the following: 
“My awareness of these teaching strategies was enhanced” (90%), “I am prepared to use these 
strategies in my teaching” (90%), “the session(s) was/were relevant to my teaching/work within the 
school” (90%), and “the information was conveyed in a way that was easy to comprehend and fol- 
low” (89%). 


e« The question with the lowest level of agreement was “I feel comfortable enough with the information 
| learned that | could share it with my colleagues,” with 81% either agreeing or strongly agreeing. 


How effectively were strategies implemented by teachers who attended the Institute? 


The effectiveness of implementation of ExcELLence Institute strategies was assessed via an online sur- 
vey completed by teachers who had attended the training sessions - this was a continuation of the sur- 
vey described above. The first six items in the survey concerned degree of difficulty faced when trying to 
implement the learned strategies in their classrooms (See Appendix E, p. 20). 


e Ease of implementation: In comparison with the questions concerning the reactions to the original 
training they received (See above), attitudes toward implementation of ExcELLence Institute strate- 
gies were less positive (79% of statements rated positive). 


« The most positive responses were to the item “things | learned during training were easily imple- 
mented in the classroom” (85%). Teachers also felt that students liked the inclusion of them in their 
classes (81%) and observed positive benefits for students after using the strategies in their class- 
room (78%). 


e Positive reaction fell off quickly after this, particularly to survey items that concerned support or as- 
sistance they had received; “other district staff facilitated my use of these strategies” (53% agree- 
ment), and “principals and other administrators facilitated my use of these strategies” (46%). 


e More than half of teachers (52%) felt that including the ExcELLence Institute strategies in their 
teaching increased their workload. 


- How often were specific strategies used: Seventeen items in the survey asked how frequently teach- 
ers used specific strategies in their classroom during the year (see Appendix F for responses, p. 
21). 


e Among the most frequently used strategies were: "model and use complete sentences" (87% 
“usually” or “always"), "randomize/rotate to call on students" (86%), "use sentence stems to develop 
language and academic vocabulary” (84%), "post and use word walls" (83%), "scaffolding using a 
gradual release model” (81%), and "implement language and content objectives" or use of response 
signals" (71%). 


e Methods used less frequently included "use of huddle" (45%), "be the lead reader" (48%), and "turn 
the light on" or "get to know me" (49%). 


e How easy was it to use specific strategies: Seventeen items also asked how easy or difficult it was 
to use the various learned strategies in their classroom (see Appendix G, p. 22). 


e Not surprisingly, whether a strategy was judged to be "very easy" or "easy" to use was related to 
how frequently it was used. The correlation between these two variables was significant (r = 0.89, p 
< .01, statistically significant at p < .05). 


e Strategies judged to be the easiest to use were "randomize/rotate to call on students" (87%) and 
"model and use complete sentences (86%). Those judged to be more difficult included "use of hud- 
dle" (66%) and "be the lead reader" or "turn the light on" (67%). 


What was the impact of ExcELLence Institute training on the academic performance of students 
in classes taught by trained teachers? 


A detailed explanation of data analysis procedures for student performance can be found in Appendix | 
(p. 24). Table 1 provides a brief summary of the various student performance measures which were an- 
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alyzed. Data from both elementary and secondary students were analyzed. Briefly, student performance 
data included results from four different standardized tests. For example, the STAAR 3-8 assessment 
included tests for mathematics, reading, science, and social studies. The performance measure ana- 
lyzed for the STAAR 3-8 was the scale score on each of these four tests. For the STAAR EOC, five sep- 
arate test results were analyzed; Algebra |, Biology, English | and Il, and U.S. History. Stanford 10 can 
be interpreted similarly. For the TELPAS, results were analyzed for reading (reading scale score) and 
yearly progress (percent of student who made gains in proficiency between 2013 and 2014). 


Detailed student performance results can be found in Appendix J (pp. 25-26). Summary results are 
shown schematically in Table 2, which should be interpreted as follows: 


e On the left side of the matrix are the four subject areas tested; mathematics, reading/language arts, 
science, and social studies. 


e At the top of the matrix are the four self-identified subject areas taught by teachers who participated 
in the ExcELLence Institute training. Each teacher selected one and only one of these core areas. 


e All results are summed across the various different assessments shown in Table 2 (STAAR, EOC, 
Stanford, etc.). 


e Each cell shows the number of different analyses which showed a significant performance ad- 
vantage for students whose teachers participated in training, compared to those whose teachers 
had never received any similar training. 


e For example, the cell representing the intersection of reading/ELA and reading (dark red) shows that 
out of the seven different measures analyzed, none showed a significant performance advantage on 
reading assessments for ESL students of trained teachers whose core area was reading. 


e Similarly, the intersection of the social studies row and the social studies column (highlighted in dark 
green) showed that students whose social studies teachers received training showed a performance 
advantage in social studies tests for two of the three tests analyzed. 


e Overall, students whose social studies teachers were trained showed the most performance gains, 
with all subject areas except reading showing some evidence for improvement (fourth column). 


- In total, 4 out of 16 separate analyses showed performance benefits for students whose social stud- 
ies teachers were trained (25%), compared to 2 out of 16 for mathematics or science teachers and 0 
out of 16 for reading/Ilanguage arts teachers. 


Table 2. Schematic Summary of Student Performance Results (see Appendix J for details) 


Teacher's Trained Subject 


Math Read/ELA Science SocStudies All Teachers 
Math 


Tested | Reading/Language/Writing 
Subject Science 


Social Studies 


e In terms of subject area, social studies assessments showed the most evidence for performance 
gains, with 6 of 15 measures analyzed showing significant benefits for students of trained teachers 
(fourth row of Table 2). 


e In summary, while there was some evidence that ExcELLence Institute training benefited student 
performance on assessments, this was mostly limited to social studies and was weak overall (only 
11 of 80 analyses showing positive effects). 


What was the impact of ExcELLence Institute training on the academic performance of students 
at campuses that were known to have implemented the strategies? 


The data overall show that there are some performance benefits for students of trained teachers, but 
that these effects are weak and dependent on both the core subject taught by the teacher and the sub- 
ject tested. An important question is whether these benefits can be shown to be greater for teachers 
who are known to have implemented and used the strategies in their classrooms. The final data analysis 
reported here summarizes results from an attempt at this type of analysis. 


Consultation with Multilingual Programs department and Professional Support and Development depart- 
ment staff suggested that there were three campuses where special efforts were made to use the strate- 
gies included in the Everyday ExcELLence Institute during the 2013-2014 school year: Hartman MS, 
Deady MS, and Sam Houston HS. At these campuses, the principals and deans attended the Excel- 
lence Institute training and there were a number of additional training opportunities offered during the 
year (e.g., early dismissals, PLCs, etc.). The training and strategies from the Institute were included as 
part of their campus improvement plans for 2013-2014, and these campuses appeared to offer the type 
of administrative support that has been noted as a weakness in previous years (See p. 6). Note that 
these three campuses were identified not by a review of student performance data, but on the basis of 
staff judgments as to how dedicated the campuses were to implementing the strategies in the class- 
room. 


Table 3. Schematic Summary of Student Performance Results (see Appendix K for details): Anal- 
yses of Data From the Three Targeted Campuses 


Teacher's Trained Subject 


Math Read/ELA Science SocStudies All Teachers 
Math 


Tested | Reading/Language/Writing 
Subject Science 


Social Studies 


e A schematic summary of results is shown in Table 3 above. This table can be interpreted in the 
same way as Table 2 (p. 8). For details see Appendix K (pp. 27-28). 


e Overall, there was more evidence for positive effects of the training, with 17 of 80 separate analyses 
showing a significant advantage for students of teachers at these campuses. 


e In particular, performance on reading/language arts tests was benefited to at least some extent re- 
gardless of the subject taught by the teacher (10 of 35 analyses showing positive effects). 


e Positive effects for reading were seen on STAAR 3-8 and on TELPAS yearly progress. 


Figure 5. STAAR reading and TELPAS progress results for Excellence Institute students at tar- 
geted campuses and other ESL students 2014: A. Adjusted scale score in STAAR reading, and B. 
Adjusted percent making gains in English proficiency (significant effects indicated) 
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e Summary data for STAAR reading and TELPAS yearly progress are shown in Figure 5. Data from 
ESL students at the targeted schools are shown in blue, and data from district ESL students whose 
teachers received no training are shown in yellow. 


e ESL students at targeted campuses showed large benefits on both STAAR reading (average 26 
scale score points) and in the percentage of students who improved their TELPAS rating in 2013 
(average of 7 percentage points), compared to students of untrained teachers. 


Figure 6. STAAR reading and TELPAS progress results for Excellence Institute at targeted cam- 


puses and other ESL students 2014: A. Adjusted scale score in STAAR reading, and B. Percent 
making gains in English proficiency (Data shown for the individual campuses) 
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e Followup analyses revealed that Hartman MS showed more robust benefits from the training (see 
Figure 6). Hartman ESL students showed large benefits on both STAAR reading and TELPAS pro- 
gress, relative to other ESL students in the district, and showed positive effects of training on 26 of 
55 comparisons across all assessments (see Appendix L, p. 29). 


e Deady ESL students showed benefits on STAAR reading but not TELPAS progress (and on only 10 


of 55 total comparisons), while Sam Houston ESL students showed only a small 5 percent benefit 
on TELPAS progress (this was the only positive effect of 60 possible comparisons). 
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Discussion 


The goal of the Everyday ExcELLence Institute training is to provide ELL teachers with practical instruc- 
tional routines that could be used with ELL students in any content area. This included an overview of 
sheltered instruction techniques, as well as eight other literacy routines. The 2013-2014 school year 
was the second year of the program. Data suggest that even though those teachers who responded to a 
survey indicated that they used the various ExcELLence Institute strategies to at least some extent in 
their teaching, the impact of training on student performance appeared modest at best. 


Even at campuses where there was, reportedly, buy-in from administrators, and a concerted emphasis 
on the use of the strategies reviewed in the Institute, there was no evidence for a general improvement 
in student performance across core subject areas. Furthermore, gains were not uniformly observed 
across all targeted campuses. Reading/language arts performance of ESL students appeared to benefit 
the most from the training, as indicated by the positive effects observed on the STAAR and on TELPAS 
yearly progress. There was also some evidence of a positive impact on social studies performance. 


Prior versions of sheltered instruction training offered by the district (Houston Independent School Dis- 
trict, 2010; 2011, 2012, 2013) found evidence for small but beneficial effects of sheltered instruction 
training on student performance. However, these benefits were not uniform across core subject areas 
and sometimes only appeared in cases where teachers were judged to have implemented the strategies 
to a high degree (Houston Independent School District, 2013). The present results would appear to con- 
tinue this pattern, which suggests that the training offered via the Institute provides some benefits for 
students, but that these are relatively small. In addition, the training does not appear to benefit students 
across content areas, which was supposedly one of the advantages of this "strategies" approach. 


Before concluding, it must be pointed out that the Everyday ExcELLence institute has developed over 
the past two years to become a much different program than was originally anticipated. When it was 
originated, the program was focused on secondary-level ELL students only, and including training on 
more than two dozen different strategies in eight categories. The Everyday ExcELLence Institute 
evolved out of this and focuses on a much smaller set of eight core strategies teachers could use. How- 
ever, it is now targeted at teachers in grades 1—12, and is also open to teachers of non-ELLs. Indicative 
of this latter trend is the fact that the title "Everyday ExcELLence" has now become "Everyday Excel- 
lence" (note the lack of capitalization for "ELL"). Given these developments, it is unclear what expecta- 
tions there should be concerning the program's impact on the district's ELL students. 
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Appendix B 
Sheltered Instruction Background Information 


Sheltered instruction is a style of teaching which makes grade-level academic content in core areas 
(e.g., math, science, social studies) more accessible for English Language Learners (ELLs), while at the 
same time promoting development of English language proficiency. It highlights key language features 
and incorporates strategies to make content more comprehensible to students, without sacrificing rigor. 
Sheltered instruction is sometimes referred to as SDAIE (specially designed academic instruction in 
English). While use of sheltered instruction techniques has come to be widespread in U.S. schools, this 
growth has often been characterized by inconsistent practices from district to district, and even from 
class to class within the same school (August & Hakuta, 1997; Berman et al, 1994; Kaufman, et al., 
1994; Sheppard, 1995; Short, 1998) 


Arguably, the most popular version currently in use is the sheltered instruction observational protocol, or 
SIOP (Echevarria & Graves, 1998; Echevarria, Vogt, & Short, 2000). The SIOP model was developed in 
a seven-year national research project (1996-2003) sponsored by the Center for Research on Educa- 
tion, Diversity, and Excellence (CREDE). Researchers identified features of instruction present in high- 
quality sheltered lessons, and developed an observational tool consisting of 30 items grouped into three 
sections: preparation, instruction, and review/evaluation. The instruction component is further broken 
down into clusters of items dealing with building background, comprehensible input, strategies, interac- 
tion, practice/application, and lesson delivery. 


All features of the SIOP model are aligned with current research on instruction for ELLs. SIOP was origi- 
nally designed to be used as an observation and rating tool for researchers, but it was soon recognized 
that the instrument could be used by teachers for lesson planning and reflection. Some of the tech- 
niques and strategies which SIOP encourages include the following: 


e use of supplemental materials, 

e adapt content to level of student proficiency, 

e link concepts to student background and experiences, 

e — link past learning and new concepts,- 

e use scaffolding techniques, 

e allow for frequent interactions between student-teacher and among students, 

e use hands-on materials or manipulatives, and 

e provide activities that integrate all language skills (reading, writing, listening, speaking). 


Research has shown that the SIOP model is effective for learners at all grade levels across many sub- 
ject areas, and can impact student achievement (Echevarria, Vogt, & Short, 2004). 


District teachers received SIOP training in 2009-2010, and two different evaluations (Houston Independ- 
ent School District, 2010; 2011) found evidence of performance gains for students whose teachers had 
received sheltered instruction training, 


Sheltered instruction training for district ELL teachers has undergone modification in the past three 
years. A significant factor in this has been the prominence of literacy issues for the district's secondary 
ELL students. Results of the 2013 NAEP reading test showed that 22% of district non-ELL students in 
grade 8 were at least proficient in reading (i.e., reading at roughly an 8th grade level or higher). Howev- 
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Appendix B (continued) 


er, only 1% of 8th grade ELL students were rated as proficient (NCES, 2014). Note that these figures 
represent a performance decline from 2011, where the corresponding values were 20% and 2% respec- 
tively. Poor reading skills constitute a significant barrier for ELL students. This is because addition to 
interfering with their ability to master course content, inadequate reading skills prevent many ELLs from 
exiting ELL status (ELLs must meet specific standards in reading, writing, and oral English proficiency in 
order to cease being classified as ELL). Accordingly, sheltered instruction training in 2011-2012 was 
augmented by including a review of various strategies meant to improve student literacy. This portion of 
training borrowed heavily from a recent review by Beers (2003). Subsequently, in 2012-2013, the Ex- 
cELLence Institute attempted an approach which simplified things by reducing the number of individual 
strategies that were taught to secondary ELL teachers, while still placing a dual emphasis on sheltered 
instruction and strategies for improving student's literacy. 
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Appendix C 


Number of Teachers and Other Staff Attending the Everyday ExcELLence Institute 


Campus 


Almeda ES 
Askew ES 
Atherton ES 
Attucks MS 
Austin HS 
Barrick ES 
Bellaire HS 
Benavidez ES 
Berry ES 
Black MS 
Bonham ES 
Brookline ES 
Browning ES 
Bruce ES 
Burbank MS 
Burnet ES 
Carrillo ES 
Chavez HS 
Clifton MS 
Coop ES 
Cornelius ES 
Crockett ES 
Cullen MS 
Cunningham ES 
Daily ES 
Davis HS 

De Zavala ES 
Deady MS 
DeAnda ES 
DeBakey HSHP 
Dogan ES 
Dowling MS 
Durkee ES 
Eliot ES 

Elrod ES 
Emerson ES 
Field ES 
Fleming MS 
Foerster ES 
Fondren MS 
Fonville MS 
Forest Brook MS 
Frost ES 
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Furr HS 

Garden Villas ES 
Gregg ES 
Gregory-Lincoln MS 
Grissom ES 
Gross ES 
Hamilton MS 
Harper Alternative 
Harris, RP ES 
Hartman MS 
Hartsfield ES 
Helms ES 

Henry MS 

Herod ES 
Herrera ES 

HS Ahead Academy 
Hilliard EL 
Hines-Caldwell ES 
Holland MS 
Jackson MS 
Jefferson ES 
Johnston MS 
Jones HS 
Jordan HS 
Kashmere HS 
Kelso ES 
Kennedy ES 

Key MS 

King ECC 

Kolter ES 

Lee HS 

Liberty HS 

Long Academy 
Longfellow ES 
MacGregor ES 
Madison HS 
Marshall MS 
Martinez ES 
McGowen ES 
McReynolds MS 
Milby HS 

Milne ES 
Mitchell ES 
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Campus 


Montgomery ES 
Neff ECC 

Neff ES 

North Forest HS 
North Houston ECHS 
Osborne ES 

Paige ES 

Park Place ES 
Patterson ES 
Pershing MS 
Petersen ES 
Pilgrim Academy 
Pin Oak MS 

Piney Point ES 
Poe ES 

Pugh ES 

Reagan HS 
Reagan K-8 Ed Center 
Revere MS 
Reynolds ES 

Rice School 

River Oaks ES 
Robinson ES 
Rodriguez ES 
Roosevelt ES 
Rucker ES 

Sam Houston MST 
Scarborough ES 
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Campus 
Scarborough HS 
Scroggins ES 
Shadydale EL 
Sharpstown HS 
Southmayd ES 
Sterling HS 
Stevens ES 
Stevenson MS 
Sugar Grove Academy 
Sutton ES 
Tinsley ES 
Twain ES 
Valley West ES 
Waltrip HS 
Washington HS 
Welch MS 
Wesley ES 
West Briar MS 
Westbury HS 
Westside HS 
Wheatley HS 
White ES 
Williams MS 
Worthing HS 
Yates HS 
Young ES 


Young Men's College Prep 


Other District Staff 
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Teachers 
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Appendix D 


Questions and Responses From Online Survey Administered to 
Everyday ExcELLence Institute Participants 


Items concerning the trainers/facilitators: 


Strongly Strongly 
Survey Item Agree Agree Neutral Disagree Disagree 
Adequately set the tone and background for 59% 31% 8% 0% 3% 
information presented in the session(s) (62) (32) (8) (0) (3) 
Allowed me to reflect and share my ideas/views 58% 32% 7% 0% 3% 
about the topics presented (61) (34) (7) (0) (3) 
Helped me to make connections with the 60% 30% 7% 1% 3% 
information so that | could use it in my teaching (63) (31) (7) (1) (3) 
Actively encouraged collaborative discussion poe oo oe OR ee 
(65) (31) (5) (0) (3) 
Items concerning the sessions themselves: 
Strongly Strongly 
Survey Item Agree Agree Neutral Disagree Disagree 
The information was relevant and useful to my 58% 28% 11% 0% A% 
daily teaching/work (61) (29) (11) (0) (4) 
The topics were well organized and paced oR Ani) ee ue ve 
(61) (29) (10) (1) (4) 
The learning outcomes for the sessions were 57% 30% 10% 0% A% 
clearly communicated (60) (31) (10) (0) (4) 
| feel comfortable enough with the information | 51% 30% 15% 1% 3% 
learned that | could share it with my colleagues (54) (31) (16) (1) (3) 
Handouts were useful and adequately supported 58% 27% 10% 2% A% 
the information presented (60) (28) (10) (2) (4) 
The information was conveyed in a way that was 55% 34% 8% 0% 4% 
easy to comprehend and follow (57) (35) (8) (0) (4) 
My awareness of these teaching strategies was 54% 35% 8% 0% 3% 
enhanced (57) (37) (8) (0) (3) 
| am prepared to use the strategies in my teaching pee can oe oe Be 
(50) (44) (8) (0) (3) 
Overall, the session(s) was/were relevant to my 57% 32% 7% 0% A% 
teaching/work within the school (60) (34) (7) (0) (4) 
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Appendix E 


Questions and Responses From Online Survey Administered to 
Everyday ExcELLence Institute Teachers Concerning the Overall Ease 
of Implementing ExcELLence Strategies in Their Classroom 


How easy was it to use the methods you learned about in the classroom? 


Strongly Strongly 
Survey Item Agree Agree Neutral Disagree Disagree 
Things | learned during training were easily 35% 51% 10% 3% 2% 
implemented in the classroom (36) (52) (10) (3) (2) 
Including the strategies in my teaching increased 25% 27% 19% 22% 6% 
my workload (26) (28) (20) (23) (6) 
| observed positive benefits for students after 41% 37% 14% 7% 2% 
using these strategies in my classroom (44) (40) (15) (7) (2) 
Students appear to like the inclusion of these 39% 43% 16% 1% 2% 
strategies in my classroom (39) (43) (16) (1) (2) 
coecliatanywccnes: =6§| 2 ah Oh OH 
Sadie (21) = (29) (34) (8) (3) 
Principals and other administrators facilitated my 27% 19% 38% 11% 5% 
use of these strategies (26) (19) (37) (11) (5) 
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Appendix F 


Questions and Responses From Online Survey Administered to Teachers 
Concerning How Often They Used Specific Strategies 


Survey Item Always Usually Sometimes Seldom Never 

27% 37% 27% 3% 7% 

Pump Up The Vocab (26) (36) (26) (3) (7) 
, 26% 38% 25% 6% 6% 
se es (25) (37) (24) (6) (6) 
17% 32% 34% 6% 3% 

Be The Lead Reader (16) (30) (32) (6) (11) 
F 28% 30% 32% 2% 7% 
Pen/cil to Paper (27) (29) (31) (2) (7) 
17% 32% 33% 6% 12% 

Turn The Light On (16) (30) (31) (6) (11) 
25% 29% 27% 7% 12% 
2 

Do | Really Get It? (23) (27) (25) (6) (11) 
21% 23% 35% 6% 14% 

Huddle (20) (22) (33) (6) (13) 
21% 28% 35% 7% 9% 

Get To Know Me (20) (27) (34) (7) (9) 
“ii 38% 33% 23% 1% 5% 

Implement Language and Content Objectives (36) (32) (22) (1) (5) 
F 35% 35% 18% 4% 7% 

Use Response Signals (34) (34) (17) (4) (7) 
Question Using Q3SA (Question, Signal, Stem, 20% 37% 24% 13% 7% 
Share, Assess) (19) (34) (22) (12) (6) 
. 29% 37% 18% 8% 8% 

Embed gestures Into Learning Process (28) (35) (17) (8) 8) 
54% 30% 12% 2% 3% 

Post And Use Word Walls (51) (28) (11) (2) (3) 
61% 26% 8% 1% 4% 

Model And Use Complete Sentences (60) (25) (3) (1) (4) 
66% 20% 10% 0% 4% 
Randomize And Rotate To Call On Students (64) (19) (10) (0) (4) 
Use Sentence Stems To Develop Language 47% 36% 10% 2% 4% 
And Academic Vocabulary (46) (35) (10) (2) (4) 
Scaffold Using A Gradual Release Model (e.g., A8% 33% 14% 3% 2% 
"| Do, We Do, You Do") (47) (33) (14) (3) (2) 
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Appendix G 


Questions and Responses From Online Survey Administered to Teachers 
Concerning How Easy or Difficult It Was to Use Specific ExcELLence Strategies 


Very Very 
Survey Item Easy Easy Neutral Difficult Difficult 

28% 39% 31% 2% 0% 

Pump Up The Vocab (27) (37) (29) (2) (0) 
: 29% 44% 24% 3% 0% 

Bele lars (27) (41) (22) 3) (0) 
20% 46% 30% 2% 1% 

Be The Lead Reader (19) (43) (28) (2) (1) 
: 32% 45% 21% 3% 0% 

Pen/cil to Paper (29) (41) (19) (3) (0) 
21% 46% 33% 0% 0% 

Turn The Light On (19) (42) (30) (0) (0) 
28% 39% 29% 4% 0% 

? 

Do | Really Get It? (25) (35) (26) (4) (0) 
28% 38% 28% 5% 0% 

a (26) (35) (26) (5) (0) 
33% 41% 26% 0% 0% 

Get To Know Me (31) (38) (24) (0) (0) 
+ ati 29% 44% 24% 2% 0% 

Implement Language and Content Objectives (26) (40) (22) (2) (0) 
: 37% 43% 19% 1% 0% 

Use Response Signals (34) (39) (17) (1) (0) 
Question Using Q3SA (Question, Signal, Stem, 25% A3% 24% 7% 1% 

Share, Assess) (23) (39) (22) (6) (1) 
F 22% 50% 26% 3% 0% 

Embed gestures Into Learning Process (20) (46) (24) (3) (0) 
48% 36% 14% 2% 0% 

Post And Use Word Walls (45) (33) (13) (2) (0) 
51% 35% 14% 0% 0% 

Model And Use Complete Sentences (46) (32) (13) (0) (0) 
53% 33% 13% 0% 0% 

RandomizeAnd Rotate To Call On Students (48) (30) (12) (0) (0) 
Use Sentence Stems To Develop Language 40% 44% 16% 0% 0% 

And Academic Vocabulary (37) (41) (15) (0) (02) 
Scaffold Using A Gradual Release Model (e.g., 37% 47% 16% 0% 0% 

"| Do, We Do, You Do") (34) (43) (15) (0) (0) 
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Appendix H 
Explanation of Assessments Included in Report 


The STAAR is a state-mandated, criterion-referenced assessment used to measure student achieve- 
ment. STAAR measures academic achievement in reading and mathematics in grades 3-8; writing at 
grades 4 and 7; social studies in grades 8; and science at grades 5 and 8. The STAARLL is a linguisti- 
cally accommodated version of the STAAR given to ELLs who meet certain eligibility requirements. 


For high school students, STAAR includes end-of-course (EOC) exams in English | and II, Algebra |, 
Biology, and U.S. History. In 2013-2014, students in grades 9 through 11 took the EOC exams. Stu- 
dents who were in grade 12 (as well as some who had been retained) took the exit-level TAKS tests in 
order to graduate, but given the small amount of data available for this group no TAKS data were includ- 
ed in the present analyses. 


The Stanford 10 is a norm-referenced, standardized achievement test in English used to assess stu- 
dents’ level of content mastery. Stanford 10 tests exist for reading, mathematics, and language (grades 
1-8), science (3-8), and social science (grades 3-8). This test provides a means of determining the rel- 
ative standing of students’ academic performance when compared to the performance of students from 
a nationally-representative sample. 


The TELPAS is an English language proficiency assessment which is administered to all ELL students 
in kindergarten through twelfth grade, and which was developed by the Texas Education Agency (TEA) 
in response to federal testing requirements. Proficiency scores in the domains of listening, speaking, 
reading, and writing are used to calculate a composite score. Composite scores are in turn used to indi- 
cate where ELL students are on a continuum of English language development. This continuum, based 
on the stages of language development for second language learners, is divided into four proficiency 
levels: Beginning, Intermediate, Advanced, and Advanced High. 
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Appendix | 
Analysis of Student Performance Data 


Student performance data was analyzed for the STAAR, STAAR EOC, Stanford 10, and TELPAS. The 
following describes some factors which guided these analyses: 


Post Hoc Design: The study used a post hoc design, where students were identified only after being 
taught by one of the teachers who received training. This meant that it was not possible to select an ap- 
propriately matched comparison group. Instead, the group of comparison students was composed of all 
other secondary ESL students whose teachers had not participated in the ExcELLence Institute. 


Analysis of Covariance: Since treatment and comparison groups could not be precisely matched, all 
analyses used an analysis of covariance procedure. In this, the students performance in 2014 was ana- 
lyzed, with their performance in 2013 serving as a covariate. In this way, student performance in 2014 
was adjusted to take into account their prior performance level. 


Teachers Who Were Previously Trained: A related difficulty concerned the fact that some version of 
sheltered instructional training for secondary ELL teachers has occurred in four previous years in the 
district. Therefore, it is not sufficient to simply compare student of teachers trained in 2013-2014 to stu- 
dents whose teachers were not trained in 2013-2014, since some of these "comparison" teachers may 
well have been trained in the use of similar strategies in previous years. Instead, students of teachers 
trained in the current year need to be compared to students of teachers who had never been trained. 
This was accomplished by recording, for each teacher who participated in the ExcELLence Institute, a 
record of how many previous training they had received, if any, between 2009-2010 and 2013-2014. 


Teacher Subject Area: Another important aspect of the analyses was that data were analyzed separate- 
ly for different groups of teachers who specialized by subject area. For example, results for mathematics 
teachers who were trained were analyzed separately from those of teachers who specialized in reading/ 
ELA. This was done because it made it easier to determine whether students benefited by having their 
teachers of specific subject areas trained in use of these ESL strategies. In addition, it potentially al- 
lowed differences between teachers across subject area to be seen, which could facilitate modifications 
to training procedures in the future. 


Covariates: For STAAR, STAAR EOC, and TELPAS reading, the data analyzed were the scale scores 
for particular subjects from 2014. The covariates used were the 2013 TELPAS reading scale score for 
that student. If a student did not have two years of results then their data were excluded from analysis. 
For the the Stanford 10, the NCE from 2014 was the dependent variable, and the NCE from 2013 was 
the covariate. All assessment subtests or subjects were analyzed separately. Finally, analyses were 
conducted using the combined data from all teachers, ignoring the specific subject they happened to 
teach. 
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Appendix J 


Results of Statistical Analyses of Student Performance Data 


Assessment Type, Subject Tested, and Teacher Subject Area 
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Appendix J (continued) 


Results of Statistical Analyses of Student Performance Data: 


Assessment Type, Subject Tested, and Teacher Subject Area 
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Appendix K (continued) 
Selected Campuses Versus Untrained Teachers 


Results of Statistical Analyses of Student Performance Data: 
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Appendix L 


Summary of Campus-Level Results for Selected Campuses: Schematic Results for 
All Possible Assessments and Test Subjects 


TEACHER SUBJECT 
Math Reading Science SocStu_ All Tchrs 


Math 
Reading 
Science 
SocStu 


TEACHER SUBJECT 
Math Reading Science SocStu_ All Tchrs 


Algebra 1 
Biology 
English 1 
English 2 
US History 


TEACHER SUBJECT 
Math Reading Science SocStu Il Tchrs 


Math 


Reading 
Language 
Science 


Soc Sci 


TEACHER SUBJECT 
Math Reading Science SocStu_ All Tchrs 


TELPAS 


Note: Letters indicate campuses where a Statistically significant benefit was observed for a particular comparison. 
H = Hartman MS, D = Deady MS, S = Sam Houston HS 
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